DOCUMENT RESUME 



ED 435 674 



TM 030 304 



AUTHOR 
TITLE 
PUB DATE 
NOTE 

PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Fusco, Dana; Sabo, Kim 

Participatory Evaluation: A Democratizing Instrument? 
1999-11-00 

8p.; Paper presented at the Annual Meeting of the American 
Evaluation Association (Orlando, FL, November 3-6, 1999). 
Reports - Evaluative (142) -- Speeches /Meeting Papers (150) 

MFOl/PCOl Plus Postage. 

*After School Programs; Chinese Americans; *Democracy; 
Educational Environment; Elementary Secondary Education; 
Immigrants ; Literacy; *Participation; *Program Evaluation 
* Participatory Evaluation; Stakeholders 



ABSTRACT 



This paper examines the overall process of implementing 
participatory evaluation methods in a nonparticipatory , nondemocratic 
environment. The paper is based on a 1-year evaluation of an after-school 
literacy program and is part of a larger effort to bring participatory 
evaluation to the grantees of a foundation. The evaluation occurred with an 
after school program that provides services to Chinese immigrant youth and 
their families, along with a creative, nontraditional learner environment for 
children between the ages of 5 and 14. A nonparticipatory and nondemocratic 
organizational climate was evident early on and reflected throughout the 
evaluation process. Teachers were not included in evaluation planning, and 
there was an evident lack of communication between the director and her 
staff. Staff meetings and staff journal writing increased the sense of 
participation, but the biggest success as transformative participatory 
evaluators was in working with the director to see the value of staff 
feedback. (SLD) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



Participatory Evaluation: A democratizing instrument? 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL 
HAS BEEN GRANTED BY 

T)o.a<3L-Fil9cO 



,TO THE EDUCATIONAL RESOURCES 
information center (ERIC) 



Dana Fusco 
York College 

The City University of New York 
Jamaica, NY 11451 
fusco@york.cuny.edu 

Kim Sabo 

Innovation Network 
New York, NY 
kimsabo@aol.com 



u.s. department of education 

OHice of Educational Research and improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

B-'Tfiis document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



• Points of view/ or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



Paper presented at the annual meeting of the American Evaluation Association, Orlando, 
Florida (November, 1999). 



This paper will examine the overall process of implementing participatory evaluation 
methods in a non-participatory, non-democratic environment. The paper is based on a ' 
one-year evaluation of an after-school literacy program and is part of a larger 
foundation-driven effort to bring participatory evaluation to its grantees. We will 
describe our initial goals and methods critically assessing the moments when our 
evaluation strategies were democratizing. The questions addressed are: Can 
participatory evaluation methods create more participation within a program? What are 
the organizational factors that need to be in place for these tools to be democratizing? 
What have we learned from the process of trying to implement participatory evaluation 
techniques in a non-democratic environment? 



Introduction 

Since 1997, the Robert Bowne Foundation has spearheaded the initiative, "Re-imagining the After 
School Program," as a catalyst for organizational development among its grantees. The foundation 
works with a cluster of New York City nonprofits that are invested in the development of their 
literacy programs and ultimately improving children’s educational outcomes. The initiative has, 
supported organizational capacity-building through staff training, mentoring, financial support, and 
an evaluation coach for each grantee to promote ongoing learning and improvement. This paper will 
examine the process of implementing participatory evaluation within one of these programs. 

There are a variety of different types and styles of participatory evaluations. However, in a recent 
edition of New Directions for Program Evaluation (1998) two streams of participatory evaluation 
were identified under which all others fall. These include Practical participatory Evaluation (P-PE) 
and Transformative Participatory Evaluation (T-PE) (Cousins and Whitmore, 1998; Brisolara, 

1998). Each of these approaches has different goals and rationales and are situated within two 
distinct philosophical positions. Practical Participatory Evaluation (P-PE) supports "programmatic 
or organizational decision making by involving stakeholders (sometimes in a limited manner) in 
certain aspects of the evaluation process" (Brisolara, 1998). P-PE grew out of practice and theory 
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developed in the United States, Great Britain, Canada, Norway and Sweden. This approach is 
rooted in Peirce’s and Dewey’s philosophies regarding the social nature of knowledge construction. 
These theoretical positions along with Lewin’s drew the scientific community’s attention to the role 
of human agency in knowledge construction. 

P-PE’s central goal is to promote the utilization of evaluation findings. It is believed that the '• 
participation of stakeholders in the evaluation process enhances the relevance and ownership of 
findings. Therefore, evaluation results are more likely to be used (Cousins & Whitmore, 1998). P- 
PE is often limited in terms of its involvement of stakeholders; it takes a conservative approach to 
action and relies on the expert status of the professional evaluator (Brisolara, 1998). Further, P-PE 
privileges empirical or scientific methods over popular modes of knowledge production. 

On the other hand. Transformative Participatory Evaluation (T-PE) is rooted in principles of 
emancipation and social justice, thereby having less conservative goals that P-PE. T-PE seeks to 
"empower members of community groups who are less powerful than or are otherwise oppressed by 
dominating groups" (Cousins & Whitmore, 1998). Further, T-PE uses participation and action as a 
means to democratize social change, creating the conditions by which participants can empower 
themselves (Cousins & Whitmore, 1998). The primary focus of T-PE is on the empowerment of 
individuals and/or groups through the process of constructing and respecting their own knowledge. 

Upon entering the field our goals and theories were in line with the transformative model. 

However, we soon came face to face with the reality that our program had not counted on making 
these types of qualitative changes. They were not interested in becoming more democratic and did 
not see this as the goal of the evaluation. Our different understandings combined with the fact that 
the evaluation was required by the foundation made our job a difficult one. Nonetheless, the 
following account demonstrates how over time, and with increased trust and candor, the evaluation 
had a democratizing effect on staff relations. 

Organizational context 

The evaluation occurred within an after-school program that provides services to Chinese 
immigrant youth and their families. The agency’s mission is to: 1) improve English language 
proficiency among recent immigrants, 2) help children and families adjust to a new way of life, 3) 
affirm Chinese cultural identity and heritage, and 4) promote integration and assimilation into 
mainstream American society. The after-school program challenges traditional ESL pedagogy, ' 
which focuses on grammar, vocabulary out of context, spelling and phonics, by providing a 
creative, non-traditional learning environment for children between the ages of 5 and 14. In addition 
to homework assistance, which meets the demands of working parents, the program seeks to 
enhance literacy development through Origami and other hands-on literacy activities. Through 
Origami students learn the ancient Chinese art while being introduced to literature and language 
activities. The program is based on the belief that language acquisition is easiest when students are 
having fun, and are not conscious of structural teaching. 

During the first year of the grant initiative, the Origami program was funded as an exploratory 
endeavor with fifteen students from one ESL class. In the second year, the program goals were to 
build on its initial success by 1) increasing the ESL enrollment to 30 children, 2) developing a 
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stronger staff base, 3) improving parental involvement in literacy through origami, and 4) 
evaluating the impact on children’s motivation and English proficiency. 

Evaluation design 

The evaluation began in the second year of the grant initiative. The general goals were: 

♦ To determine the extent to which the program was operating and providing the services 
stated in their model 

♦ To understand to what extent and how the anticipated outcomes have come to pass 

♦ To uncover any unanticipated outcomes 

♦ To support the program staff to integrate evaluation strategies into their everyday 
practice 

♦ To support democratic practice 

The following discussion presents the methods within a historical framework and critically reflects 
on the moments when our evaluation strategies were and were not democratizing. The participatory 
nature of the evaluation allowed the methods to continuously emerge throughout the period of the 
evaluation as relationships were built, staff changed, program improvements were implemented, 
and new interests arose. 

Initial stakeholder meetings 

The overall evaluation design was decided upon during initial meetings with the program staff. : 
Initial meetings with stakeholders were meant to explore staffs interests and questions about their 
program. What did they want to learn about their program and their students? What did they .think 
was important? We worked with staff to identify the program’s short and long-term objectives and 
the strategies implemented in reaching those objectives. From these discussions we developed a 
Program Flow Chart, and devised a set of possible evaluation questions. These included questions 
about both project implementation and outcomes. For instance: Was the program providing the 
services as stated in their work plan? To what extent had the teachers’ attitude and approach , 
changed since the beginning of the year? Has students’ interest in learning to read and write 
increased? Have students become more proficient at speaking, reading, listening and writing 
English? In short, staff interests, program objectives and strategies, and existing evaluation methods 
were channeled into a livable and usable plan for the current evaluation. ' , 

During these initial discussions, disagreement among staff emerged. While the staff was interested 
in measuring learning and changes in learning habits, the director felt that interest in learning was 
both more measurable and more likely to be a direct result of Origami and others hands-on 
classroom activities. The director won and activities, such as those provided by the bilingual 
counselor, were initially deerhed outside the scope of the current evaluation. Further, English 
proficiency was considered an intermediate goal and not expected to change drastically during the 
first year. 

The evaluation, then, focused primarily on student interest. We decided that monthly classroom 
observations would provide the best source of data for assessing students’ interest in Origami and 
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other literacy activities. Since there was some concern that the ESL teacher was new to the program 
and observations might not yield much information, the ESL teacher was also asked to keep a 
journal, recording the activities that seemed to best excite the children. Further, we would conduct 
two interviews with the teacher to show whether teacher training changed her attitude and approach, 
and hold two focus groups with the students "to determine student’s changes in attitudes and interest 
over time." In short, the initial evaluation 1) focused on the ESL class, 2) was aimed at assessing 
student interest, and 3) incorporated observations, journal writing, interviews, and focus groups. 

Unfortunately, we were unable to include the teachers in the initial stakeholder meetings. The ■ 
executive director did not want to pay for this and did not feel that the teacher would be helpful in 
this process. Upon reflection we might have been more insistent about the teacher’s involvement. 
However, we were trying to build a trusting relationship with the executive director and did not 
want to push her on this issue. 

Changes in methodology 

Approximately six months into the project, changes were made in the scope and focus of the 
evaluation. The initial focus on students gave way to a focus on teachers. These changes reflected 1) 
the limited participation of the ESL teacher in journal writing and the lack of data accessible from 
the ESL class, 2) the introduction of professional development workshops in January, and 3) the 
increased candor of the director with the evaluation team leading to new discussions and 
possibilities. 

As relationships were built and trust was developed in the evaluation process and the evaluation 
team, the director became increasingly candid about her perceptions of the ESL teacher and the 
program overall. Part of what facilitated this process was the director’s understanding that the 
evaluation team was not there to judge her or the program but rather to facilitate a reflective process 
that might ultimately help improve the program and organization. 

With increased trust, we learned that the director seemed very concerned about teacher 
development. The director also revealed that she had a method in place for understanding the , 
strengths and weaknesses of her 13 instructors and could "systematically" place each instructor in 
one of three tiers. These tiers seemed based on the director’s understanding of good pedagogy as 
student-centered, active, engaging and creative. Teachers that did little to engage their class in 
interesting and creative activities were deemed least qualified. In discovering the director’s informal 
evaluation strategy we began to discuss the possibility of systematizing her method and possibly 
documenting the process by which teachers move from one tier to the next. Further, the professional 
development workshops offered an occasion for seeing change arriong the teachers. While initially 
the director was not interested in a "comparative" evaluation, she became more attracted to the 
possibility of involving other instructors in the evaluation as a way to help the program improve. 

We began to implement journals and focus groups with all instructors. 

Journal writing 

Eight of the thirteen instructors were receiving professional development through the foundation 
initiative, including the ESL teacher. The idea of journal writing was introduced to the group and 



O 

ERIC 



5 



i 



Participatory Evaluation: A democratizing instrument? -- Page 5 



seven of the eight began to keep journals of their classroom activities, one teacher incorporating the 
views of her students in the journals. One teacher not in the training also kept a journal. 
Unfortunately, journal writing occurred sporadically and we had many difficulties collecting the 
journals from staff members. In this way, we had little support to institutionalize this approach from 
either the director or the educational coordinator. On the other hand, when we collected the journals 
at the end of the semester we found that teachers had written in the journals more frequently than 
had been expected. One teacher documented her classroom practices at least once per week and 
often twice per week from April to June. In her journal, she included the writings of both her 
students and her assistant. Another teacher used a similar approach but only documented three ' 
lessons. A third teacher used the journals over a five-week period. Other teachers wrote in the 
journals less frequently, i.e., documenting from one to three lessons. 

Focus groups with teachers 

In March, we held the first of two focus groups with the instructors. This was a turning point in the 
evaluation and its value to the program. The focus groups not only allowed the evaluation team to 
better understand the program, what was occurring, and the impact it was having on teachers and 
students, but was itself a tool that allowed the instructors to come together, socialize, and learn from 
other’s experiences. Further, the focus group (and our interim report based on it) was the impetus of 
increased communication between the director and the instructors. The director acted on the 
recommendation made in the first interim report, which read as follows. 

Recommendation: Monthly group meetings with the classroom instructors (or anonymous 
monthly surveys) would yield continuous information about the errors and successes of 
various classroom activities. This information on a continuous basis would create a 
constant feedback loop for informing future programmatic decisions. For instance, this 
type of data may help develop staff training workshops geared around issues that emerge 
or may help in selecting classroom materials. Further, a forum where instructors can voice 
their ideas and suggestions and develop strategies for program improvement will help 
create a democratic environment whereby people work collectively to help each other 
develop and to help the program develop. 

The director used the feedback from the focus group to hold the first staff meeting and to begin 
incorporating the instructors’ ideas into new program activities, such as a gardening project. In turn 
the instructors felt more a part of the overall program. The program director and staff were finally 
beginning to value our work and were beginning to see how we might be useful. 

Revised Evaluation Questions 

About mid-way through the seven-month evaluation, the ESL teacher was transferred to a different 
classroom, a new ESL teacher was hired, and the educational director quit. These changes had an 
enormous impact on the evaluation process and we needed to regroup quickly. In the end, we 
limited our evaluation questions to those closest to the literacy program, eliminating questions 
regarding community and parent involvement. These questions were considered too difficult to 
address in this first year of the evaluation. The outcome measures focused on staff development, 
student interest and to a lesser extent, English proficiency. 
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Sharing the final report 

Our final report included attendance/enrollment records for the number of ESL children 
participating; student interest through teacher reports, classroom observations and a student survey; 
and English proficiency. The report also focused on the development of the teachers and the 
changes resulting from staff turnover. The director was presented with the results and during a 
meeting invited the bilingual counselor to help interpret the data — a switch from her earlier 
resistance to his ideas. The counselor added crucial insights and the data was the impetus to a rich 
discussion about the philosophical disagreements between staff and parents, as well as the role that 
the evaluation team might play next year in helping to create a method for feedback to the parents 
on their children’s progress. 

Conclusion 

Evaluations are often conducted in hope that the findings might inform and help improve programs. 
Rarely is the evaluation itself and its impact discussed. In this concluding section we re-visit how 
this participatory evaluation created democracy within a non-democratic after-school program and 
discuss the lessons we learned along the way. 

The non-democratic and non-participatory organizational climate was evident early on and reflected 
throughout our evaluation process. Teachers were not invited to attend initial stakeholder meetings, 
and few knew of the evaluation until we conducted the first focus group. While the bilingual 
counselor and educational coordinator were invited to attend the initial meetings their views were 
quickly discounted. The lack of communication between the director and her staff was also evident. 
There were no consistent staff meetings and few opportunities for the instructors to get together and 
share resources and experiences. The instructors often complained that they did not know when 
trainings were occurring and that they were not informed when children would be taken out of their 
classes for counseling or other activities. 

The executive director expressed to us that the instructors had limited abilities and were not 
committed to teaching. That she did not see them as valuable to the evaluation, then, was not 
surprising. Yet, it took a while for us to recognize a discrepancy. What we found was that the ■ 
teachers were enthusiastic about their work and many were very interested in developing their 
instructional skills — the shear number of volunteers wanting to participate in the professional ’ 
development sessions proved this. Eight out of the thirteen instructors were involved in these • 
trainings every two weeks and during these sessions were engaged and enthusiastic. Others 
expressed an interest in attending the workshops but did not know about the training sessions in' 
advance. Further, all of the staff came to both of opr focus groups and eight kept journals. 

In many ways, it seemed that their level of interest and conunitment to teaching had been 
underestimated and likely related to overall organizational climate. However, over the course of the 
evaluation we witnessed a resurgence of energy and activity in the organization as a whole with 
increased staff, new program activities, and the inclusion of the instructors in decision making. 
Following our first focus group the executive director held a meeting with the staff to discuss many 
of the issues raised during the focus group. During this meeting, the executive director began to 
realize the importance of coming together as a group and that the teachers had good ideas and 




7 



Participatory Evaluation: A democratizing instrument? -- Page 7 



suggestions. In sharing the final report, the director also for the first time that we observed, 
explicitly asked for the feedback of her counselor. In these ways, the evaluation process had a 
democratizing impact on the program. Namely, the evaluation resulted in opening lines of 
communication and giving greater latitude to staff in making programmatic decisions. 

A second mechanism for increasing democracy in the program was through journal writing. 
Through the journals, it was evident that teachers were trying new things and working on new skills 
in the classroom. One teacher who was not in the teacher training volunteered to do journal writing. 
While we did not ask her for student feedback, she chose to include the student’s voices. 
Interestingly, it seemed that this became both a way for her to get feedback from the children and a 
literacy activity. Every day different children gave feedback in the journal and the teacher would 
correct their spelling and punctuation, working directly with the children on literacy skills. In this 
way, the evaluation itself seemed to support her to develop a new literacy activity as well as 
increase students’ voice in the program. 

In summary, it seemed that our biggest success as "transformative participatory evaluators" was in 
working with the director to see the value of staff feedback. She went from understanding her 
instructors as young, uncommitted, under trained youth to viewing them as having something to say 
about literacy in the classroom. While this shift did not fully democratize the program, it certainly 
allowed significant shifts in communication to occur between the director and her staff, including 
the teachers and counselors. The role that the journals played in opening up a dialogue between at 
least one teacher and her students was also evident; however since this was not a focus of the 
evaluation less can be said about its democratizing effect. One could presume, for instance, that this 
teacher was already including students’ feedback in her methods but the journal became a new tool 
for achieving this goal. 

In attempting to utilize a participatory evaluation approach, we have learned the importance of 
recognizing a rion-democratic environment and the need to explicitly address the objective of 
supporting democratic practice. For instance, we might consider in the future recommending a 
quorum for stakeholder meetings. We have learned about the need to achieve buy-in for a 
democratic approach, or at least collaboratively and explicitly define the objectives and terms. We 
have learned the importance of building trust and candor, particularly with those holding 
organizational power, while simultaneously articulating disagreements and discrepancies. Finally, 
we have learned that change takes time and does occur - slowly and gradually, and in small, albeit 
not unimportant, ways. 
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